MADPH-01-1230 

WM-01-107 

hep-ph/0106277 

June, 2001 

Search for t — > ch at e + e Linear Colliders 



Tao Han* and Jing Jiang^ 

Department of Physics, University of Wisconsin, 
1150 University Avenue, Madison, WI 53706, USA 

Marc Sher* 

Nuclear and Particle Theory Group, Physics Department, 
College of William and Mary, Williamsburg, VA 23181, USA 



Abstract 

We study the rare top-quark decay t — > ch, where h is a generic Higgs boson, 
at a linear collider. If kinematically accessible, all models contain this decay 
at some level due to quark flavor mixing. Some models, such as Model III 
of the two-Higgs doublet model, have a tree-level top-charm-Higgs coupling, 
and the branching ratio is close to 0.5%. Others, such as the MSSM, have 
a coupling induced at one- loop, and can have a branching ratio in the range 
of 10~ 5 — 5 x 10~ 4 . We find that a linear collider of a/s = 500 GeV and a 
luminosity of 500 fb _1 will begin to be sensitive to this range of the coupling. 
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I. INTRODUCTION 



Two of the most important unanswered puzzles of elementary particle physics are the na- 
ture of electroweak symmetry breaking and the origin of flavor. Both are implemented in the 
Standard Model of particle physics in an ad hoc manner. Often, extensions of the Standard 
Model (SM) treat these two problems separately; extended Higgs models do not address the 
flavor problem and flavor symmetry models do not substantively address the nature of elec- 
troweak symmetry breaking. Yet, there are no dynamical models satisfactorily addressing 
the two puzzles coherently. Since the top quark is so heavy, with a mass on the order of 
the electroweak symmetry breaking scale, it is tempting to consider the possible special role 
of the top quark in the electroweak symmetry breaking sector and in the quark flavor sec- 
tor, and its interactions could shed light on both problems. Flavor-changing-neutral-current 
(FCNC) decays of the top quark H could be very sensitive to various extensions of the 
Standard Model that address both of these puzzles. 

Most studies of flavor-changing-neutral-current top-quark decays have examined the de- 
cay of the top quark into a charmed quark and a gauge boson H (either a gluon, photon 
or a Z). However, the flavor-changing- neutral-current decay most likely to shed light on 
the nature of electroweak symmetry breaking is the decay of the top quark into a charmed 
quark and a Higgs boson (or the decay of the Higgs boson into a top and a charmed 
quark if kinematically preferred). It is this interaction that we study in this paper. 

All models will contain a top-charm-Higgs (TCH) vertex at some levelQ. Even in the 
Standard Model, a one-loop diagram gives a contribution, but it is GIM-suppressed and 
extremely small ||. In extensions of the Standard Model, however, such a vertex can occur 
either at tree level or at a non-GIM-suppressed one-loop level. 

One of the simplest extensions of the Standard Model has a tree-level TCH vertex. 
This is the general two-Higgs doublet model, referred to as Model III, in which no discrete 
symmetries are imposed to avoid flavor- changing neutral currents. In this case, the quark 
Yukawa coupling matrices consist of two parts, y = yi + y2, where yi is the coupling to the 
i'th Higgs doublet. Since diagonalizing y will not, in general, diagonalize y\ and y2, there will 
be tree-level flavor-changing neutral currents. Cheng and Sher |JJ] noted that if one wishes 
to avoid fine-tuning, then the observed structure of the mass and mixing hierarchy suggests 
that the FCNC vertex £y cjiqjh is given by the geometric mean of the Yukawa couplings of 
the quark fieldsQ 



1 Whether a TCH vertex exists does, in part, depend on nomenclature. Strictly speaking, one 
should consider the "Higgs" to be the other member of the isodoublet containing the longitudinal 
components of the weak vector bosons; after all, it is this field that is associated with spontaneous 
symmetry breaking. With that definition, the Yukawa couplings are necessarily flavor-diagonal. 
Here, we use the word "Higgs" to indicate a scalar field whose mass eigenstates contain some part 
of the field associated with spontaneous symmetry breaking. 

2 We note that some authors may have used a different normalization for Ajj by a factor of y/2 
larger than ours. Since the ansatz is order of magnitude only, either definition is acceptable. 



2 



Cij = \j ~ > (1) 

where v = 246 GeV is the weak scale, and the A^- are naturally all of 0(1). This ansatz has 
no conflict with current phenomenological observation, and yet predicts rich physics in the 
heavy-flavor generations. Under this ansatz, the top-charm-scalar coupling £ ci is the biggest 
in the model, and is approximately 0.05. 

Model III is fairly general. A number of more specific models have a large TCH coupling 
as well. Burdman || studied a topcolor model 0, which gave a large TCH coupling of 
the same order as Model III (although his "Higgs" boson had a mass larger than that of 
the top quark). A topflavor model analyzed the flavor-changing coupling of the top 
and charm to a top-pion, finding a TCH coupling that is substantially larger than that of 
Model III. Recently, a bosonic topcolor model was proposed that automatically has the 
TCH coupling as the largest fermionic coupling of the Higgs boson. An extensive analysis 
of models with additional singlet quarks (Q) by Higuchi and Yamamoto |9|] gives a flavor- 
changing coupling which is proportional to the product of the cQ and tQ mixings; this can 
be larger than the Model III coupling. In general, in models which treat the top quark 
differently than other quarks motivated by its weak-scale mass, one might expect TCH 
coupling of Model III size or larger. 

There have been numerous studies of Model III with the above ansatz, examining 
K, B, jj, and r decays, as well as K°, B° and D° mixing |TIJ. Here, however, we will 
be concerned with the large TCH coupling only. 



The rate for the decay t — > ch was first calculated by Hou [TT| . For a top-quark mass of 
175 GeV, the branching ratio of t — > ch to t — > bW + is 

T(t -> ch) _ a2 m, (1 ~ ^? 



T(t -> bW+) ct m t (i _ ^) (i m w 2 
w 0.009 A 




This implies that the branching fraction of t — » ch can be typically of 10~ 3 or higher for 
\ ct ~ 0(1). If the Higgs is heavier than the top, one can have h — > ic + ct, as first discussed 
in Ref. [O]. This leads to the process e + e~ —* h°A° — > ttcc + cctt at a linear collider 
|j~2||, to like-sign top-pair production at the LHC and to the linear collider process 



e + e — > is e v e (tc + ct) [|TJ]. These processes all assume a Higgs boson heavier than the top, 
however, and recent electroweak precision data [pjj, as well as hints from LEP [ 16| , prefer 



the scenario of a rather light Higgs boson, making t — > ch kinematically accessible. As one 
can see from Eq. (0), the branching ratio for X ct = 1 and a 115 GeV Higgs mass is 3 x 10~ 3 . 

If one looks at one-loop processes, there are many more possibilities. Most extensions of 
the Standard Model will avoid the GIM suppression that makes the coupling so small in the 



Similarly, we have not worried about the running mass effects at different renormalization scales, 
and we simply take the pole masses for mt and m c for our analysis. 
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FIG. 1. Tree level Feymann diagrams for e + e — ► tch process. Diagrams for e + e — > tch 
process are similar. 



Standard Model M. The most popular extension is the MSSM. In this model, there must be 
a Higgs boson lighter than the top quark, and so the decay will occur. An extensive analysis 
of t — > cH, where H refers to any one of the three neutral scalars in the MSSM, was carried 
out in Ref. [fL7|| . They show that the dominant contribution comes about from loops with 



gluinos and squarks, and the branching ratio ranges from 10 -5 to 5 x 10~ 4 over the MSSM 
parameter space. In an R-parity violating SUSY model, a one-loop contribution can also 
give |18|] a branching ratio as large as 10 -5 . 



We see that models with a tree-level TCH coupling have branching ratios of the order 
of 10" 3 — 10~ 2 , and models with a one-loop coupling can have branching ratios of the order 
of 10~ 5 — 10~ 4 . What level is experimentally detectable at high energy colliders? Recently, 
discovery limits at the LHC were calculated |20| and are approximately 5 x 1CT 5 . While 
the results are quite encouraging, detailed Monte Carlo simulation would be needed to draw 
a definitive conclusion due to the complicated background issue at hadron colliders. In a 
linear collider, backgrounds are much more manageable, although signal cross sections are 
lower. In this paper, we assume that a light Higgs boson will be observed and examine in 
detail the discovery potential for its FCNC coupling at a linear collider. 



II. TOP-CHARM-HIGGS COUPLING AT A LINEAR COLLIDER 

A. Production Cross Section 

Using the coupling constant given in Eq. (|l|), we wish to explore the limits obtainable 
on A c t at a linear collider. We consider the process 

e + e~ — > ich, tch , (3) 

where the corresponding Feynman diagrams are shown in Fig. |1| The dominant contribution 
to process (|3]) is from e + e~ — > ti followed by t (i) decay into ch(ch), namely <r(e + e _ — > 
ti) x [Br(t —>■ ch) + Br(i — > ch)]. The decay branching ratio is calculated and shown in 
Fig. 0(a), where we see that for = 120 GeV and A^ = 1, t — > ch has a branching ratio 
of 2.8 x 1(T 3 . At tree level, top -pair production has a total cross section of 580 fb at the 
center-of-mass energy ^Js = 500 GeV. Thus each channel will have a cross section of 1.6 
fb. The total cross section of e + e~ — > ich and tch is presented in Fig. |2|(b) as a function of 
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FIG. 2. (a) Branching ratio of t — * ch decay as a function of and (b) Cross section for 
tch + ich production as a function of center-of-mass energy in e + e~ collisions for rrih = 120 GeV. 



y/s, where and henceforth = 120 GeV is chosen for illustration. The total cross section 
scales as A5L and is of order of 1 fb for A ct = 1. 



B. Signal And Background 



Since we are motivated to consider a light Higgs boson with a mass around 120 GeV, 
we concentrate on the process in which the Higgs boson decays to bb and t decays into bW. 
The W can then undergo either hadronic (2 jets) or leptonic {^v) decays. The h — > bb 
branching ratio for nth = 120 GeV is about 85%. The signal thus consists of the following 
channels in the final state 

e + e~ -^bb 1 b + 3jets (4) 
or ->66, b£ ± u + I jets. (5) 

The primary background to the above signal is from the SM top-pair production 
e + e~ — > tt — > bW + bW~ , with one W decaying hadronically and the other decaying ei- 
ther hadronically or leptonically, depending on which signal of Eqs. (£|) and ([|) we are 
looking at. In order to identify the signal from the background, we require that 3 6's be 
tagged. The efficiency for a single b tagging is taken to be 65% fi~9fl . This still does not 
eliminate the background: For the case that both W's decay hadronically, one out of the 
four non-6 jets from W decay to light quarks may be misidentified as a 6-jet. We assume 
the misidentification to be 1% for each jet. Thus for four jets the total misidentification 
probability is 4%. Similarly, for the case that one of the two W's decays leptonically, one 
of the two non-6 jets from W decay to light quarks is misidentified as a 6-jet. The SM 
background induced by one W decay into cb is negligible due to the small size of the CKM 
mixing matrix element ~ 0.04. The entries of the first row in Table | show the signal 
and background cross sections at y^i = 500 GeV before applying any kinematical cuts but 
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signal 


bg(hadronic) 


bg(leptonic) 


bg(total) 


no cuts 


0.75 


6.56 


3.23 


9.79 


basic cuts 


0.60 


0.28 


0.26 


0.54 


addi. cuts 


0.52 


0.13 


0.12 


0.25 



TABLE I. Cross sections in fb at y/s = 500 GeV for signal (m^ = 120 GeV and X c t = 1) and 



hadronic, leptonic and total backgrounds before and after cuts. 



including decay branching fractions and 6-tagging efficiencies. We find that the background 
rate is still overwhelming. 

We next try to make use of the distinctive signal kinematics by reconstructing h, t, i 
and W masses from the final state momenta. To simulate the detector response, we smear 
the jet energies according to a Gaussian spread 

where © denotes a sum in quadrature. For the signal, we first consider the case where W 
decays into two jets. Of the three tagged 6's, we choose two 6's whose invariant mass is 
closest to rrih and identify them as the 6's coming from h decay (M£ ec ). Then from the 
three light jets we pick the jet that, together with 66 selected above, produces an invariant 
mass (M t rec ) that is closest to m t . This jet is now identified as c from t — > ebb decay. The 
remaining two jets can be used to construct the W mass M^ c and with the third b, we have 
M[ ec . If W decays into lu, it is simpler. We simply need to choose two of the three 6-jets to 
construct M£ ec . The only non-6 jet is identified as c. We get M^ c from the momenta of £ ± 
and missing energy assigned to u, after adding the momentum of the third b, we have M\ ec . 
For the background in which W decays hadronically, we randomly pick one from the 4 non-6 
jets and assume it to be misidentified as a 6-jet. Then we follow the same construction as 
for the signals, we will get M£ ec , M£ ec , Mp c and M^ c . For the background when W decays 
leptonically, we randomly choose one of the two non-6 jets to be misidentified as 6 and 
again follow the analysis for the signals to get the four reconstructed invariant masses. The 
normalized differential distribution with respect to these reconstructed masses are shown 
in Fig. |3] for signals and backgrounds. The characteristic difference between the signal and 
background is evident from these mass distributions. The first crucial reconstructed mass 
is MJl ec . A peak structure in this variable provides the confidence of the signal observation 
and gives the discrimination power between the signal and background. The wide width 
is due to the jet-energy smearing discussed earlier. The other distinctive mass variable is 
M[ ec = Mbbc, which reconstructs m t for the signal and no structure for the background due 
to the incorrect jet combination. Similarly, M[ ec w m t for the signal but it spreads out 
for the background, as seen in Fig. 0(c). Those distributions motivate us to device more 
judicial kinematical cuts. Our basic kinematical cuts are listed in Table [TI| based on these 
four reconstructed masses. 

Additional cuts can be applied to further increase the signal-to-background ratio. We 
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FIG. 3. Normalized distribution for the signal (solid), hadronic background (dashes) and lep- 
tonic background (dots) with respect to the reconstructed masses (a) M^ ec , (b) M[ ec , (c) M£ ec 
and (d) M^f, with = 500 GeV and m h = 120 GeV. 



first notice that the charm-jet energy from the top decay is monotonic in the top-rest frame 
as a result of a two-body decay 



rprest _ m t (-> m h\ (7 \ 

which is about 50 GeV for rrih = 120 GeV. In contrast, the faked charm jet from W decay will 
have an energy more spread out. The normalized distributions are shown in Fig. [|. Figure 
U(a) is the E c distribution in the e + e~— lab frame which presents a nearly clear range due to 
the Lorentz boost of the top quark motion, where the sharp end-point in E c is sensitive to 
m/j, that may provide an independent kinematical determination for m^. Figure ^(b) shows 
the E c distribution in the top-quark rest frame based on the t — > bbc reconstruction. We 
see that E c is monotonic near 50 GeV modulus to the jet-energy resolution. This provides 
a good discriminator for the signal and background. In fact, the reconstructed Higgs boson 
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basic cuts additional cuts 



110 < M r h ec < 130 20 < E l c ab < 130 

160 < M[ ec , Mf ec < 190 40 < E r c est < 60 



itmx 



65 < M$° < 95 E™ n < 120, 80 < E\ 

TABLE II. The basic and additional kinematical cuts ap- 
plied to our signal-to-background optimization (all values in 
GeV). 



energy in the top-quark rest frame should be also monotonic E]^ st = ^(1 + m^/m^), but it 
is correlated with E r c est . We also look at the two 6-jets coming from h decay and examine 
the harder and softer ones of the two in energy, separately, as illustrated in Figs. f|(c) and 
f|(d). The optimal cuts are given in Table f|. After applying the basic and additional cuts 
the signal-to-background ratio improves significantly, as summarized in the last two rows of 
Table § 



C. Sensitivity to the tch coupling 

Given the efficient signal identification and substantial background suppression achieved 
in the previous section, we now estimate the sensitivity to the FCNC tch coupling from this 
reaction using Gaussian statistics, which is applicable for large signal event samples. We 
define the statistical significance by 

No 

a = . 3 :, (8) 
y/N s + N B ' V ; 

with Ng and Ng being the number of signal and background events. We note that a = 3 
(called 3a) approximately corresponds to a 99% confidence level. Figure |5] presents the 
3a (2a) sensitivity to the FCNC couplings by the solid (dashed) curve as a function of 
integrated luminosity for ^Js = 500 GeV and = 120 GeV. Recall that at X c t = 1, the 
t — > ch branching ratio is about 2.8 x 10~ 3 . Such an order of magnitude can be anticipated 
in models with tree-level FCNC couplings, and can be easily observed at a LC with an 
integrated luminosity of less than 40 fb -1 . As X c t varies, the branching ratio scales like 
2.8 x 10 _3 A^ t . For 1-loop induced FCNC decays such as in SUSY models, the branching 
ratios can be about 10~ 5 — 5 x 10~ 4 , corresponding to A ct of 0.06 — 0.4. A linear collider 
with 500 fb -1 integrated luminosity will begin to be sensitive to this range of the coupling 
of A ct ~ 0.4 (0.3) at a 3cr (2a) level, but a higher luminosity will be needed to extend the 
coverage of the parameter space to a level about 0.2. 
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III. CONCLUSIONS 

Models with a top-charm-Higgs coupling come in two categories: Those with a tree- 
level coupling and those with a coupling induced at one-loop. In the former case, such 
as the Model III two-Higgs doublet model, one expects A ci to be of 0(1). If the Higgs is 
somewhat lighter than the top quark, then the t —>■ ch decay will be detected within the 
first 40 fb" 1 of running of a linear collider, as indicated in Fig. |. In the latter case, such as 
supersymmetric models (in which we know that one of the Higgs bosons will be sufficiently 
light), the branching ratios can be in the range of 10~ 5 to a few times 10 -4 . This corresponds 
to a X ct of 0.06 — 0.4. A linear collider with 500 fb _1 integrated luminosity will begin to be 
sensitive to this range of the coupling at a 3a level, and higher luminosity will be needed to 
substantially extend the coverage of the parameter space. 
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FIG. 5. 3cr sensitivity (solid) to the FCNC tch couplings at *Js = 500 GeV for m h = 120 GeV 
as a function of integrated luminosity. The dashed curve is for 2a sensitivity. 
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